Existing security event correlation systems collect data from a number of distributed security agents into a central server where the data is correlated. As the amount of data collected increases, the burden on the system can become excessive. The burden can consume network bandwidth in communicating events to the central server, burden the central server processor with correlating all of the events, and increase disk or memory storage burden on the central server from storing all the events.
Generally, methods to resolve overloading of correlation systems include filtering, aggregating, or otherwise preprocessing events at the distributed agents. By preprocessing data at the agent, the total number of events that must be centrally correlated can be reduced to a manageable level. However, any such preprocessing introduces the possibility that valuable information will be removed at the agents and not be available for accurate correlation at the central server. Additionally, correlation rules being enforced at the central server take any such preprocessing into account, and may have to be manually modified if the agent preprocessing is reconfigured.
Thus, it is advantageous to allow correlation rules for a central server to accommodate distributed architecture, without modifying the rules. Further, it is advantageous to allow individual agents to store events locally and initially report only the most significant events, to reduce the consumption of network bandwidth. It is then possible to store all potentially supporting data in a distributed fashion, to be retrieved as needed. For example, such an approach would allow the vast majority of common firewall events to not be reported—only those that are needed to corroborate an attack detected by an intrusion detection system would ever be retrieved.